Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffBrad Leblanc Date: Jan 14, 2006 1:04am
Forum: etree Subject: Information about XML (metadata) files and Deriver

Hi folks, in the interest of educating our patrons, curators thought it would be a good idea to start documenting and communicating some of the more technical information about the contribution system. It is currently a slightly complex system and takes some getting used to. At some point, we’ll probably find a permanent and easier to find home for this information, but for now we think posting it to the forum is better than not sharing it.

In this post we will cover the "XML" files, the "rules.conf" file and how they relate to lossies getting created and the deriver.

Stage 1:
When you import a new recording, you begin by filling out an "import form" to describe the concert. In this form you will receive the option of disallowing lossy derivatives. We call them derivatives because the original lossless FLAC or SHN files are used to "derive" the Ogg and MP3 files that you see on many of the recordings. The engine that performs this work is aptly titled the "deriver". The option of allowing/disallowing lossies presents itself at the bottom of the import screen as seen in attached image1.jpg.

Occasionally you may notice that this option has been disabled due to the artist’s request that lossies *never* get created. (see image2.jpg)

When you submit this first form, it will create 2 files from the info you have provided:

1. [itemID]_meta.xml (mandatory for the item to continue processing)
2. [itemID]_rules.conf (optional for the item to continue processing – only created if the person uploading the recording or the artist opt out of having lossy files)

Where [itemID] = the name of the folder you used for the recording being imported.

The [itemID]_meta.xml file contains all the information you submit in the form - including source, lineage, taped by, transferred by, setlists etc… This information is used to "paint" the details page that other patrons will browse in the future.

The [itemID}_rules.conf file will only be created if you opt-out of the lossy formats. This is done by checking the box we discussed above. The content of this file is the text string "CAT.lossy" which tells the deriver not to create lossy files, and only create the full show zip.

Stage 2:
Following the submittal of the above form is the form called "File Options" in which you describe all of the files you have contributed. -- Is the file a FLAC file or SHN? Is it CD Quality or 24bit audio? Is the file text? If the file is a music file, give the song a title and the artist a name. If the file is text information then you should identify the author in the "creator" field – usually the taper’s name. If the file is music, the "creator" field should auto fill with the name of the artist in the recording. Submit this information when you have confirmed all of it is accurate to the best of your knowledge. See image3.jpg for a copy of this form.

This information creates another XML file:

1. [itemID]_files.xml (mandatory for the item to continue processing)

If you have allowed lossy files to be created, the information you entered into this form will be used to write ID tags on the lossy files. This means that media players such as iTunes, WinAmp, Windows Media Player etc… will show artist name, album info, and song names automatically when the files are played. However, this is a 1 shot deal – if you enter the information here incorrectly, updating it after the lossy files are created will not update the tags on these files. The incorrect tags will remain until the derivatives are regenerated, so it is very important to get this right the first time.

Stage 3:
Once the system notices that both the mandatory files exist - [itemID]_files.xml and [itemID]_meta.xml - (it sweeps for them about once every 10 minutes), it will move the item on to the next stage of the process – the deriver. The files will be copied to a "worker" machine which will do the following:

- verify that both mandatory metadata files exist and the information is valid (no illegal characters)
- verify the file integrity of the audio (run included MD5’s, FLAC checks)
- check for presence of [itemID]_rules.conf
- create lossy files if no [itemID]_rules.conf was found.
- tag lossy files
- copy all finished files back to the upload server
- change the status to "Waiting for curator approval".

This post was modified by Brad Leblanc on 2006-01-14 09:04:51

Attachment: image1.jpg
Attachment: image2.jpg
Attachment: image3.jpg

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffTyler Date: Jan 8, 2006 12:18pm
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

Thanks for the info brad. Always good to see it laid out like that for users to see how the system works.

Reply to this post
Reply [edit]

Poster: Gedit Date: Jan 11, 2006 11:51am
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

nice work brad. thanks !

Reply to this post
Reply [edit]

Poster: jgiordani Date: Jun 18, 2007 9:17pm
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

I'm a newbie, and unclear on how I can get the XML file into iTunes to have ID3 info for the lossy files I download and add to my iTunes. I've tried File -> Import, and select the XML file, but that doesn't seem to do the trick. Thanks.

Reply to this post
Reply [edit]

Poster: valparaisorock Date: Jul 15, 2007 8:49am
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

Hi!
I have a question:

Is it feasible to edit the XML file's metadata and upload it through FTP?

Becasue I would like to change the name of the songs in the player and I have understood that the songs' titles come from the XML file.

Please advise and thanks in advance.
Sebastian

Reply to this post
Reply [edit]

Poster: Mr Puddintame Date: Feb 29, 2008 5:37pm
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

Step 3- tag lossy files is not working. I have uploaded files as a .wav, and the mp3 files generated by the deriver are NOT being tagged with title and artist information.

Reply to this post
Reply [edit]

Poster: Mr Puddintame Date: Mar 3, 2008 1:26pm
Forum: etree Subject: Re: Information about XML (metadata) files and Deriver

Even if the file is uploaded as a tagged 128 kbps file, a 64 kbps file downloaded from the Internet Archive will be stripped of those tags. Is anyone reading this?

This post was modified by Mr Puddintame on 2008-03-03 21:26:53